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ABSTRACT 

For the past 20 years, the University of Hawaii 
Community Colleges have had a formal process for the review of 
educational programs, requiring that each program undergo a 
systematic review at least once every 5 years. The result tended to 
be a lengthy document that chronicled the history of the previous 5 
years and was far more descriptive than analytical. If the evaluation 
process is truly to affect program planning and operations, the 
review structure should provide for program differences in selecting 
evaluation measures; data should be current and reflect immediate 
issues; the review process should be manageable and involve program 
faculty and directors; and the review findings should be presented in 
an easily understood report. Kapiolani Community College has worked 
toward developing an alternative approach that satifies these 
criteria. The cornerstone of the process is the use of program health 
indicators. For each program, faculty, administrators, and the 
institutional researchers identify four to six key evaluation 
measures, generally involving aspects of program demand and 
efficiency, satisfaction, and outcomes. For each indicator, a green 
line, which represents a satisfactory performance level, and a red 
line, which indicates that serious intervention is necessary, are 
established. The area between the red and green lines creates a sort 
of caution zone, where further analysis, observation, or minor 
changes are indicated. Additional information is provided to allow 
the reader to understand the scope and context of the program. The 
evaluation report, not to exceed 5 to 10 pages, includes a program 
description, a program health indicator graph, linear graphs, tabular 
data, narrative analysis, and appendices. Problems encountered in 
pilot testing the review process included difficulties in setting 
realistic standards for health indicators, ineffective data systems, 
data-hungr3/ faculty, low response rates not resulting in 
statistically significant results, and the currency of the data. A 
sample program review for Kapiolani *s Associate Degree Nursing 
Program is provided, excluding appendices. (MAB) 
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For the past twenty years the University of Hawaii Community 
CoUeffes have had a fomial process for the review of educational programs. 
S procesMSed and adopted by the Board of Regents, required that 
ea^h^roS-^ imdergo a systematic review at least once every five years to 
deterS^ if there was a continuing need for the program and whether the 
pro^Xwas continuing to serve its students and potential employers well. 

This process followed an outHne that called for the review to 
document the following: 

I . Organization of the program review process 
li. Piistory and description of the program 
III Program performance and analysis 

IV.' Planned program changes and project program performance 

The program analysis section required responses to eight questions: 

What is the community and State need for the program? 
Is the curriculum appropriately organized to meet program 

Is\he program appropriately organized to meet its objectives? 
Is the level of program activity appropriate, relative to student 
demand for the program, and employer demand tor its 

A^e^pr^o^iam resources sufficient to meet program objectives? 
b How efficiently does the program utiHze available resources? 
i How effective is the program in meeting its objective 
fi' Are the academic and student support services - counseling, 
advising, library, media, learning assistance, clerical sei-vices, 
etc. - and physical facilities adequate to support the program 
requirements? 

The result of the review was a lengthy (often 50-to-75 page) document that 
tended to chronicle the history of the past five years and be fa^ more 
descrintive than analytical. The reviews were viewed as a task to be 
SetTd to cTmply ^th the policy rather than of much value to the actual 
m^aSment^^^^^^ of the program. The task or completmg the 

Sview often fell to junior administrators or jumor faculty, as a sort of 
Td^^strative hazing or right of passage. It was clear that a different 
"^^Ztrl^^ng pro-ams was needed if we truly wanted to have the 
analysis impact the operations and planning for the program. 

In trying to determine the nature of an improved system, we decided 
on several critfria that we would like to see included m any design. These 
included: 

1) Different programs do not necessarily need the same data. Too 
often, the fixed format reports of the old review process reqmred 
programs to report and analyze data that had marginal or no 



1. 
2. 

3. 
4. 



5. 
6. 



meaning for the program. For example, a program in nursing 
where graduation is a requirement to sit for the licensure 
examination should monitor its graduation rates closely. A 
program in culinary arts, where the actual attainment of the 
associate degree has little economic value (students are hired 
based on specific culinary skills), m^y be less concerned or not 
concerned at all with graduation rates. The review structure 
should provide for these differences in the selection of measures 
for the program review and not blindly require the same 
information of all programs. 

2) The data should be current and reflect the issues facing the 
program today. One of the major shortcomings of the old review 
process was that it was a mismatch with the way decisions are 
made in managing programs. The once-every-five-year review 
does not represent how faculty make decisions. They do not wait 
for five years, analyze the data, and then proceed to implement 
program changes. Faculty are tinkerers. They are constantly 
adjusting their programs and the curricula to reflect small 
changes in industry demand, in industry practice, and in 
student demand and characteristics. The review process has to 
be designed to provide information that supports this form of 
incremental decision making. 

3) The review process had to be manageable. Recognizing that the 
first two assumptions were leading us toward a review process 
that was much more firequent (ideally, annually), a significar ' 
criteria was that the new process had to be workable. The notion 
that the cumbersome, 50-page reviews could be done annually or 
even biannuaJly was not seriously considered. The new 
mechanism had to be operational within the staffing limits of a 
one-person institutional research oflice and faculty and 
department chairs continuing to do their normal duties. 

4) The review process had to involve program faculty and program 
directors. Since the major purpose of this new review process 
was to impact the decision-making at the program level, the 
involvement of the program faculty in all aspects of the process 
was seen as essential. The primary decision was that the design 
would be driven by the needs of the program in managing the 
program and not by the needs of broad institutional research 
reports or end-of-the-year reports. This requires the 
involvement of faculty in both design and analysis, while 
technical aspects of data collection and preparation could be left 
to the institutional research staff. 

5) The presentation had to be quickly and easily xmderstood. We 
did not want the review process to result in reports that were 
difficult or tedious to read. We wanted instead dociunents that 
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woTild be easy to understand, quick to convey to the readers the 
strengths and weaknesses of the program, and stimulating in 
discussing what directions programs should go. This meant far 
more reliance on graphics than text and relegation of non- 
essential information to appendices or oblivion. 

With the assistance of Peter Ewell of the National Center for Higher 
Education Management Systems, Kapi'olani Community College has 
worked toward developing an alternative approach that satisfies these 
criteria. The cornerstone of the process is the use of program health 
indicators - a name drawn from its analogy with the health care field. 
When an individual goes for a physical examination, there are generally a 
few basic tests that provide the data to the physician on which to base the 
health of the individual. If those tests show data that are within normal 
guidelines, then the individual is judged OK and told to come back next 
year. If the test results fall outside the normal ranges, then the patient is 
either coimseled about improvements or subjected to additional tests to try 
to determine the extent of the problem. The more serious the deviation, the 
more serious the intervention. 

A similar process is designed into the new program review process. 
For each program, faculty, working with administrators and the 
institutional researcher, identify four to six key measures of their 
programs. These measures generally involve aspects of program demand, 
program efficiency, satisfaction, and outcomes. The exact choices of the 
indicators are program-specific (although several programs choose the 
same indicators). There is no requirement that a program have a specific 
indicator although it is essential that all data definitions remain the same. 
Program A may choose as one of its measures the performance of students 
on a licensure examination. Program B, lacking such an examination, 
may choose a different outcome measure such as student job placement 
rates. 

For each indicator, again in conjunction with administrators and 
institutional research, the faculty determine two levels of performance. 
The first level, conveniently called the green line, represents the value of 
the indicator above which everyone is satisfied. The second level, called the 
red line, represents the value of the indicator where there is serious 
concern and significant intervention is likely. The area between the green 
Une and red line creates a sort of caution zone, where further analysis, 
observation, or minor changes are indicated. For example, if nursing 
selects as one of its indicators student perfonnance on the Hcensure 
examination, they may decide that a pass rate of 85% is acceptable (green 
line), 70% is totally imacceptable (red line), and anything between 70% and 
85% is worth worrying about to see what may be contributing to a potential 
problem. 

The choice of these levels may also vary from program to program. 
Program A may choose a much higher graduation rate than Program B if 



graduation is a key to seeking employment. The choices are also not fixed 
in time. A progi^am on the rebound may set a criteria for student 
enrollment reflecting a targeted growth of 50 new students while that same 
program when stable may be satisfied with only 40 new students per year. 
Each of these choices - the health indicators themselves and the criterion 
levels that represent satisfaction with performance - are driven by the needs 
and goals of the program and involve both faculty and management input. 

Each year, the four to six health indicators are plotted graphically 
side by side (see Graph 1). A line graph overlays these indicators and 
depicts where the program falls relative to its stated goals. If a program 
generally falls within the green zone, then little analysis or follow-up is 
needed or done. If a program has several indicators within the yellow zone, 
then more analysis is needed and additional institutional research may be 
conducted. If a program has several indicators within the red zone, then 
the program is likely in serious trouble, and a full-scale intervention and 
analysis is indicated. 

In this way, the energies of the institution can be focussed on those 
programs needing the most help. Neither the administration institutional 
research, nor the faculty have to spend time doing extensive riake-work 
reports for programs that are essentially healthy. Instead, that time and 
effort can help to address problem areas as they begin to appear in a 
program. It is important to understand that the health indicators are not 
intended to be decision tools by themselves. There is no automatic 
assxmiption that any program falling within the red zone should be 
immediately eliminated or phased out. Instead, the health indicators serve 
as a triggering mechanism to invoke a response when something may be 
wrong with a program. 

While the health indicators are the heart of this system, there is 
additional information provided to the faculty and to the reader of the 
review. This additional information is included to allow the reader to 
imderstand the scope and context of tL ^ program and perhaps to provide 
some cues as to why certain indicators may have the values they do. In all 
cases, though, the additional information must conform to the conditions 
that the report be brief and readable. The full report is outlined as follows: 

I . Description of program and program goals. Limited to one page 
in length and designed to give the reader a context for the 
information that follows. 

II. Program Health Indicator Graph. Limited to one page and 
presented in color to illustrate clearly the state of tiie program. 

III. Linear Graphs. Generally, two to three data elements that are 
uresented in graph format covering the last six years of data for 
that element. This provides some historical context and trend 
data for certain measures like enrollment, applications. 
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graduation rates. These measures may or may not be the same 
as those included in the health indicators graph. 



IV. Tabular Data. Limited to one page and including some data 
elements not reflected in either the program health indicators or 
the linear graphs. The data may also present the actual values 
of the data elements in some of the health indicator data. 

V . Narrative Analysis and Recommendations. This section is 
completed by faculty in the program and may range in length 
from one page to several pages depending on the ^'health'* of the 
program and the depth of analysis that must be completed. Tliis 
section also allows the faculty to interject any additional 
information from sources not included in the normal database. 
An example of such information might be changes in State law 
that will generate curricular changes in a particular program. 

Appendices. The appendices include descriptive information such as 
the history of the progrgim, the faculty, the adviso: ^ committee 
members, the curriculum, and the definitions of the ke^ data 
elements, pH as reference for the reader. 

The complete review is then no more than 5 to 10 pages with a heavy 
reliance on graphics, a length and style that we feel is manageable for even 
an annual review. A sample of a complete review for the Associate Degree 
Nursing Program (without appendices) is included as an appendix to this 
paper. 

The process of the annual review generally follows this timetable: 

9/1 - 10/31 Institutional Research collects data based on Fall 
enrollment, previous-year graduation and transfer 
rates, etc. 

9/1 - 10/31 Institutional Research conducts surveys based on the 
survey schedule. This includes surveys of employers, 
current students, leavers, and advisory committee 
members. Not all surveys are conducted each year. 

9/1 - 10/31 Program personnel make modifications to descriptions 
to reflect changes from the previous year. 

11/1-11/30 Institutional research inputs data and prepares health 
indicator charts, trend graphs, and data tables. 

12/1 Institutional research forwards information to program 

with suggested areas of analysis. 



12/1-1/31 Program personnel complete analysis and narrative 
with review by Dean. 

2/1-3/31 Additional research conducted by institutional research, 
if needed. 

4/1 All reviews completed with recommendations for 

chginges for the next year. 

This review process has been piloted for three programs and is now in its 
first year of full operation. This early pilot effort is promising but has also 
indicated some problem areas with implementation (some of which are still 
unresolved). These include: 

1) the initial determination of the health indicators can be a time- 
consuming task and must involve both faculty and 
administrative perspectives. Most faculty have not considered 
\vhat measures they really need as they manage their programs 
and even less frequently what level of performance on those 
measures would be satisfactory to them. They will sometimes 
set too high a standard - "we want 100% pass rates on this 
examination" or too low - "we will be satisfied if 25% of the 
students are satisfied". After discussion and reassurance that 
the process is designed to help manage the program, standards 
can be agreed upon that are satisfactory to both management 
and the faculty. 

2) the current data systems do not always provide the data in 
formats or ways that are useful and necessary for good program 
management. This has required us to design special routines to 
extract and manipulate data to make it relevant to the program 
indicators the faculty have chosen. In some instances, the 
indicators chosen do not yet have data available and totally new 
data collection mechanisms have had to be designed and 
established. 

3) the faculty are data himgry. We have found that most faculty 
are eager to get information about their students and the 
students* performance. This has often resulted in requests for 
information and data that is far in excess of what we can 
manage. There has had to be a process of negotiating on what is 
a reasonable amoimt of information that can be provided. 

4) the response rates on survey instruments for small programs do 
not result in statistically significant results. We have begun to 
explore ways to increase response rates through phone follow-up 
or to consider other means such as focus groups to obtain 
feedback on student and employer satisfaction for small 
programs. 



ERLC 



8 



5) the currency of the data. To make the report most useful, 

current year data are provided* However, for some measures, 
current year data are not available at tlie time the report is 
finalized. Examples of these measures are graduation rates and 
cost data. In the end, the r-eport presents a combination of two 
years of data. Care should be given to dating the statistics used. 

It will take some time to resolve these problems and to determine if the new 
process is indeed manageable and does provide the faculty with better 
information in managing their programs. 



SAMPLE PROGRAM REVIEW 
(Without Appendices) 

KAPIOLANI COMMUNITY COLLEGE 

ASSOCIATE DEGREE NURSING PROGRAM 

1991-92 



DESCRIPTTON 



The Associate Degree Nursing program prepares students to work as registered nurses in 
hospitals, chnics, and a variety of health settings. Students are admitted to the program 
after one semester to one year of pre-requisite course work in the basic sciences and general 
education. Licensed practical nurses may also enter the second year of the program after 
completing a refresher course. 

Tlie nursing program itself is a rigorous two years in length, including summers. Students 
learn through lectures, on-campus laboratories, and extensive clinical experience in local 
hospitals. After graduation, students must pass the registry examination before being 
allowed to practice nursing. 

The Associate Degree Nursing program is one of the most popular programs at Kapiolani. 
There is a severe shortage of nurses and this shortage, coupled with rising salaries, have 
attracted students in large numbers. It is anticipated that the nursing shortage in Hawaii 
will continue throughout the 1990's. 

The program at Kapiolani is relatively new and the first students were admitted in Spring 
1989. It has grown rapidly in response to a State initiative to reduce the criticsd nurse 
shortage and will reach its capacity of 200 to 220 students by 1992. The Associate Degree 
Nursing program is one of the offerings of the Nursing Department, which also offers 
programs for licensed practical nurses, nurse aides, adult residential care operators, and 
home health aides. 

Students completing the program may also transfer to the University of Hawaii - Manoa 
and work toward a B.S.N, degree. Tlie Iavo degree programs are articulated so that 
students transfer with junior standing and can complete their B.S.N, with two years of 
additional study. 



G OAL 



To provide students with entry level skills and knowledge to function effectively as a 
registered nurse. 



HEALTH INDICATORS - NURSING 




PROGRAM TREND DATA 




ADMISSION DATA - ASSOCIATE DEGREE NURSING 

1 60 X 




89-90 90-91 91-92 
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PROGRAM TREND DATA 
GRADUATES - ASSOCIATE DEGREE NURSING 




PROGRAM DATA SUMMARY 
PROGRAM: NURSING (ADN) SEMESTER: FALL 1991 



1 . PROGRAM DEMAND 



Total Number of Applicants: 


128 


b'L ti Major Students: 


138 


Total Qualified Applicants: 


52 


FTE Other Students: 


0% 


New Students: 


40 


% Special Population: 


20.0% 


Continuing Students: 


140 






Total Maiors: 


180 








Current Work Force: 


7,500 


Total Openings/Y ear: 


1,200 


Projected Work Force: 


8,700 


New OpeningsA' ear: 


300 


2. PROGRAM EFFICIENCY 


Classes Taught: 


56 


Average Class Size: 


10 


# of Small Classes: 


5 


% Small Classes: 


8.9% 


Total Semester Hours: 


246 


% Sem Hr by Lecturer: 


0% 


Total Student Sem. Hours: 


2,072 


FTE Students/FTE Fac. 


: 8.6 


Cost per Student Sem. Hour: 


$231 






3 . PROGRAM OUTCOMES 


Credit Earned Ratio (Major): 


90.6 


Standard = 75.0 




Credit Earned Ratio (Remedial): NA 


Standard = 60.0 




Credit Earned Ratio (Gen. Ed.): 87.5 


Standard = 65.0 




% Retention of New Majors: 


90 


Standard = 50.0 




Degrees Awarded: 


32 






Graduation Rate (Fed): 


NA 






Graduation Rate (UH): 


75% 






Student Transfer Rate: 


14% 






% Grads Employed in Field: 


98.3% 


Standard = 70.0 




Licensing Rate: 


85.0% 






Graduate Satisfaction Rate: 


4.7 






Leaver Satisfaction Rate: 


NA 







PROGRAM ANALYSIS AND RECOMMENDATIONS 

The nursing program has generally perfornied above the expected levels. There are no 
major problems facing the program at this time. 

An analysis of the student success rate on die Ucensure examination, which was lower than 
desirable, reveals diat many of the faHures were among students who were completmg the 
program after being admitted as Hcensed practical nurses rather dian as fiiU two yeai^ 
students. It is unclear yet whether this is die result of die LPN students bemg out of school 
for some time and diertfore not as familiar widi die content of die first year courses, ^ 
whether diis is a one time problem, or whedier diere is some odier unknown cause. It is 
recommended that an analysis be done of the areas of the t^^st on which 
students did poorly and whether that is related to the curriculum sequencing 
taken by LPN admlttees. It is further recommended that this performance 
be monitored in future years and that LPN graduates be encouraged to 
participate in licensure preparation courses. 

Discussion widi local hospital personnel indicate tiiat diere is beginning to be a slight 
improvement in the severe shortage for registered nurses in die State. It is 
recommended that this trend should be monitored to see if the demand will 
continue to drop in the years to come. 

There is now a significant surplus in die number of qualified students and die number of 
students diat can be admitted each term. The quahfied students who are turned down for 
admission after investing one to two years in preparatory course work are likely to be very 
upset. It is recommended that a committee be formed to examine the 
admission issues and consider the initiation of a pre-nursing major that 
were provide for guaranteed admission for qualified students. 

The nursing program is applying for initial accreditation from die National League of 
Nursing in 1992 This may result in curricular or odier changes bemg recommended in 
response to die accreditation report and site visit Final action on die accreditation is 
expected in January 1993. 
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ABSTRACT 

For the past 20 years, the University of Hawaii 
Community Colleges have had a formal process for the review of 
educational programs, requiring that each program undergo a 
systematic review at least once every 5 years. The result tended to 
be a lengthy document that chronicled the history of the previous 5 
years and was far more descriptive than analytical. If the evaluation 
process is truly to affect program planning and operations, the 
review structure should provide for program differences in selecting 
evaluation measures; data should be current and reflect immediate 
issues; the review process should be manageable and involve program 
faculty and directors; and the review findings should be presented in 
an easily understood report. Kapiolani Community College has worked 
toward developing an alternative approach that satifies these 
criteria. The cornerstone of the process is the use of program health 
indicators. For each program, faculty, administrators, and the 
institutional researchers identify four to six key evaluation 
measures, generally involving aspects of program demand and 
efficiency, satisfaction, and outcomes. For each indicator, a green 
line, which represents a satisfactory performance level, and a red 
line, which indicates that seriouf intervention is necessary, are 
established. The area between the red and green lines creates a sort 
of caution zone, where further analysis, observation, or minor 
changes are indicated. Additional information is provided to allow 
the reader to understand the scope and context of the program. The 
evaluation report, not to exceed 5 to 10 pages, includes a program 
description, a program health indicator graph, linear graphs, tabular 
data, narrative analysis, and appendices. Problems encountered in 
pilot testing the review process included difficulties in setting 
realistic standards for health indicators, ineffective data systems, 
data"-hungry faculty, low response rates not resulting in 
statistically significant results, and the currency of the data. A 
sample program review for Kapiolani 's Associate Degree Nursing 
Program is provided, excluding appendices. (MAB) 
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For the past twenty years the University of Hawau Community 
rolleffes have had a formal process for the review of educational programs. 
?his process iStiated and adopted by the Board of Regents, required that 
Sroiam SSdergo a systematic review at least once every five years to 
deterSinhere was a continuing need for the program and whether the 
proS^ was continuing to serve its students and potential employers well. 

This process followed an outline that called for the review to 
document the following: 

I . Organization of the program review process 
li. Ilistory and description of the program 
III Program performance and analysis 

IV. Planned program changes and project program performance 

The program analysis section required responses to eight questions: 

1 What is the community and State need for the program? 
2". Is the curriculum appropriately organized to meet program 

Is\he'program appropriately organized to meet its objectives? 
Is the level of program activity appropriate, relative to student 
demand for the program, and employer demand tor its 

£e^pr^o^iam resources sufficient to meet program objectives? 
b How efficiently does the program utilize available resources? 
7* How effective is the program in meeting its objective ! 
8* Are the academic and student support services - counsehng, 
advising, library, media, learning assistance, clerical sei-vices, 
etc. - and physical facihties adequate to support the program 
requirements? 

The result of the review was a lengthy (often 50-to-75 page) document that 
tended to chronicle the history of the past five years and be far more 
descrintive than analytical. The reviews were viewed as a task to be 
SXed to cTmply ^th the policy rather than of much value to the actual 
mZgrraent and op^ of the program. The task for completing the 
^2^^ew often fell to junior administrators or junior faculty, as a sort of 
ad^^strative hazing or right of passage^ It was f -/^at a d« 
approach to reviewing progi-ams was needed if we truly wanted to have the 
analysis impact the operations and planning for the program. 

In trying to determine the nature of an improved system, we decided 
on several Sitlria that we would like to see included in any design. These 
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included: 



1) Different programs do not necessarily need the same data. Too 
often, the fixed format reports of the old review process required 
programs to report and analyze data that had marginal or no 



meaning for the program. For example, a program in nursing 
where graduation is a requirement to sit for the Hcensure 
examination should monitor its graduation rates closely. A 
program in culinary arts, where the actual attainment of the 
associate degree has little economic value (students are hired 
based on specific ciilinary skills), may be less concerned or not 
concerned at all with graduation rates. The review structure 
should provide for these differences in the selection of measures 
for the program review and not blindly require the same 
information of all programs. 

2) The data should be current aiid reflect the issues facing the 
program today. One of the major shortcomings of the old review 
process was that it was a mismatch with the way decisions are 
made in managing programs. The once-every-five-year review 
does not represent how faculty make decisions. They do not wait 
for five years, analyze the data, and then proceed to implement 
program changes. Faculty are tinkerers. They are constantly 
adjusting their programs and the curricula to reflect small 
changes in industry demand, in industry practice, and in 
student demand and characteristics. The review process has to 
be designed to provide information that supports this form of 
incremental decision making. 

3) The review process had to be manageable. Recognizing that the 
first two asstunptions were leading us toward a review process 
that was much more frequent (ideally, annually), a significant 
criteria was that the new process had to be workable. The notion 
that the cumbersome, 50-page reviews could be done annually or 
even biannually was not seriously considered. The new 
mechanism had to be operational within the staffing limits of a 
one-person institutional research office and faculty and 
department chairs continuing to do their normal duties. 

4) The review process had to involve program faculty and program 
directors. Since the major purpose of this new review process 
was to impact the decision-making at the program level, the 
involvement of the program faculty in all aspects of the process 
was seen as essential. The primary decision was that the desigr 
would be driven by the needs of the program in managing the 
program and not by the needs of broad institutional research 
reports or end-of-the-year reports. This requires the 
involvement of faculty in both design and analysis, while 
technical aspects of data collection and preparation could be left 
to the institutional research staff. 

5) The presentaHon had to be quickly and easily imderstood. We 
did not want the review process to result in reports that were 
difficult or tedious to read. We wanted instead doctmients that 
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wolild be easy to understand, qiiick to convey to the readers the 
strengths and weaknesses of the program, and stimulating in 
discussing what directions programs should go. This meant far 
more reliance on graphics than text and relegation of non- 
essential information to appendices or oblivion. 

With the assistance of Peter Ewell of the National Center for Higher 
Education Management Systems, Kapi'olani Community College has 
worked toward developing an alternative approach that satisfies these 
criteria. The cornerstone of the process is the use of program hedth 
indicators - a name drawn from its analogy with the health care field. 
When an individual goes for a physical examination, there are generally a 
few basic tests that provide the data to the physician on which to base the 
health of the individual. If those tests show data that are within normal 
guidelines, then the individual is judged OK and told to come back next 
year. If the test results fall outside the normal ranges, then the patient is 
either counseled about improvements or subjected to additional tests to try 
to determine the extent of the problem. The more serious the deviation, the 
more serious the intervention. 

A similar process is designed into the new program review process. 
For each program, faculty, working with administrators and the 
institutional researcher, identify four to six key measures of their 
programs. These measures generally involve aspects of program demand, 
program efficiency, satisfaction, and outcomes. The exact choices of the 
indicators are program-specific (although several programs choose the 
same indicators). There is no requirement that a program have a specific 
indicator although it is essential that all data definitions remain the same. 
Program A may choose as one of its measures the performance of students 
on a licensure examination. Program B, lacking such an examination, 
may choose a different outcome measure such as student job placement 
rates. 

For each indicator, again in conjunction with administrators and 
institutional research, the faculty determine two levels of performance. 
The first level, conveniently called the green line, represents the value of 
the indicator above which everyone is satisfied. The second level, called the 
red line, represents the value of the indicator where there is serious 
concern and significant intervention is likely. The area between the green 
line and red line creates a sort of caution zone, where further analysis, 
observation, or minor changes are indicated. For example, if nursing 
selects as one of its indicators student performance on the licensure 
examination, they may decide that a pass rate of 85% is acceptable (green 
line), 70% is totally imacceptable (red line), and anything between 70% and 
85% is worth worrying about to see what may be contributing to a potential 
problem. 

The choice of these levels may also vary from program to program. 
Program A may choose a much higher graduation rate than Program B if 



graduation is a key to seeking emplo3n3ient. The choices are also not jSxed 
in time. A program on the rebound may set a criteria for student 
enrolbnent reflecting a targeted growth of 50 new students while that same 
program when stable may be satisfied with only 40 new students per year. 
Each of these choices - *^^he health indicators themselves and the criterion 
levels that represent satisfaction with performance - are driven by the needs 
and goals of the program and involve both faculty and management input. 

Each year, the four to six health indicators are plotted graphically 
side by side (see Graph 1). A line graph overlays these indicators and 
depicts where the program falls relative to its stated goals. If a program 
generally falls within the green zone, then little analysis or follow-up is 
needed or done. If a program has several indicators within the yellow zone, 
then more analysis is needed and additional institutional research may be 
conducted. If a program has several indicators within the red zone, then 
the program is likely in serious trouble, and a full-scale intervention and 
analysis is indicated. 

In this way, the energies of the institution can be focussed on those 
programs needing the most help. Neither the administration, institutional 
research, nor the faculty have to spend time doing extensive make-work 
reports for programs that are essentially healthy. Instead, that time and 
effort can help to address problem areas as they begin to appear in a 
program. It is important to understand that the health indicators are not 
intended to be decision tools by themselves. There is no automatic 
assumption that any program falling within the red zone should be 
immediately eliminated or phased out. Instead, the health indicators serve 
as a triggering mechanism to invoke a response when something may be 
wrong with a program. 

While the health indicators are the heart of this system, there is 
additional informatfon provided to the faculty and to the reader of the 
review. This additional information is included to allow the reader to 
imderstand the scope and context of the program and perhaps to provide 
some cues as to why certain indicators may have the values they do. In all 
cases, though, the additional information must conform to the conditions 
that the report be brief and readable. The full report is outhned as follows: 

I . Description of program and program goals. Limited to one page 
in length and designed to give the reader a context for the 
information that follows. 

IL Program Health Indicator Graph. Limited to one page and 
presented in color to illustrate clearly the state of the program. 

IIL Linear Graphs, Generally, two to three data elements that are 
presented in graph format covering the last six years of data for 
that element. This provides some historical context and trend 
data for certain measures like enrollment, applications. 
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graduation rates. These measures may or may not be the same 
as those included in the health indicators graph. 



IV. Tabular Data. Limited to one page and including some data 
elements not reflected in either the program health indicators or 
the linear graphs. The data may also present the actual values 
of the data elements in some of the health indicator data. 

V. Narrative Analysis and Recommendations. This section is 
completed by facidty in the program and may range in length 
from one page to several pages depending on the ^^health'* of the 
program and the depth of analysis that must be completed. Tliis 
section also allows the faculty to interject any additional 
information from sources not included in the normal database. 
An example of such information might be changes in State law 
that will generate curricular changes in a particular program. 

Appendices. The appendices include descriptive information such as 
the history of the program, the fa ^Ity, the advisory committee 
members, the curriculum, and the uefi litions of the key data 
elements, ?n as reference for the reader. 

The complete review is then no more than 5 to 10 pages with a heavy 
reliance on graphics, a length and style that we feel is manageable for even 
an annual review. A sample of a complete review for the Associate Degree 
Nursing Program (without appendices) is included as an appendix to this 
paper. 

The process of the annual review generally follows this timetable: 

9/1 - 10/31 Institutional Research collects data based on Fall 
enrollment, previous-year graduation and transfer 
rates, etc. 

9/1 - 10/31 Institutional Research conducts surveys based on the 
survey schedule. This includes surveys of employers, 
current students, leavers, and advisory committee 
members. Not all surveys are conducted each year. 

9/1 - 10/31 Program persormel make modifications to descriptions 
to reflect changes from the previous year. 

11/1-11/30 Institutional research inputs data and prepares health 
indicator charts, trend graphs, and data tables. 



12/1 



Institutional research forwards information to program 
with suggested areas of analysis. 



12/1-1/31 Program personnel complete analysis and narrative 
with review by Dean. 

2/1-3/31 Additional research conducted by institutional research, 
if needed. 

4/1 All reviews completed with recommendations for 

changes for the next year. 

This review process has been piloted for three programs and is now in its 
first year of full operation. This early pilot effort is promising but has also 
indicated f^ome problem ai^eas with implementation (some of which are still 
unresolved). These include: 

1) the initial determination of the health indicators can be a time- 
consuming task and must involve both faculty and 
administrative perspectives. Most faculty have not considered 
what measures they really need as they manage their programs 
and even less frequently what level of performance on those 
measures wotdd be satisfactory to them. They will sometimes 
set too high a standard - "we want 100% pass rates on this 
examination** or too low - "we will be satisfied if 25% of the 
students are satisfied". After discussion and reassurance that 
the process is designed to help manage the program, standards 
can be agreed upon that are satisfactory to both management 
and the faculty. 

2) the current data systems do not always provide the data in 
formats or ways that are useful and necessary for good program 
management. This has required us to design special routines to 
extract and manipulate data to make it relevant to the program 
indicators the faculty have chosen. In some instances, the 
indicators chosen do not yet have data available and totally new 
data collection mechanisms have had to be designed and 
established. 

3) the faculty are data hungry. We have found that most faculty 
are eager to get information about their students and the 
students' performance. This has often restdted in requests for 
information and data that is far in excess of what we can 
manage. There has had to be a process of negotiating on what is 
a reasonable amount of information that can be provided. 

4) the response rates on survey instrxmients for small programs do 
not resxilt in statistically significant results. We have begun to 
explore ways to increase response rates through phone follow-up 
or to consider other means such as focus groups to obtain 
feedback on student and employer satisfaction for small 
programs. 
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5) the currency of the data. To make the report most useful, 

current year data are provided. However, for some measures, 
current year data are not available at tlie time the report is 
finalized. Examples of these measures are graduation rates and 
cost data. In the end, the report presents a combination of two 
years of data. Care should be given to dating the statistics used. 

It will take some time to resolve these problems and to determine if the new 
process is indeed manageable and does provide the faculty with better 
information in managing their programs. 



SAMPLE PROGRAiM REVIEW 
(Without Appendices) 

KAPIOLANI COMMUNITY COLLEGE 

ASSOCIATE DEGREE NURSING PROGRAM 

1991-92 



DRSCRIPTTON 

The Associate Degree Nursing program prepares students to work as registered nurses in 
hospitals, clinics, and a variety of health settings. Students are admitted to the progi'ain 
after one semester to one year of pre-requisite course work in the basic sciences and general 
education. Licensed practical nurses may also enter the second year of the program after 
completing a refresher course. 

Tlie nursing program itself is a rigorous two years in length, including summers. Students 
learn through lectures, on-campus laboratories, and extensive clinical experience in local 
hospitals. After graduation, students must pass the registry examination before being 
allowed to practice nursing. 

The Associate Degree Nursing program is one of the most popular programs at Kapiolani. 
There is a severe shortage of nurses and this shortage, coupled with rising salaries, have 
attracted students in large numbers. It is anticipated that the nursing shortage in Hawaii 
will continue throughout the 1990's. 

The program at Kapiolani is relatively new and the first students were admitted in Spring 
1989. It has grown rapidly in response to a State initiative to reduce the critical nurse 
shortage and will reach its capacity of 200 to 220 students by 1992. The Associate Degree 
Nursing program is one of the offerings of the Nursing Department, which also offers 
programs for licensed practical nurses, nurse aides, adult residential care operators, and 
home health aides. 

Students completing the program may also transfer to the University of Hawaii - Manoa 
and work toward a B.S.N, degree. Tlie two degree programs are articulated so that 
students transfer with junior standing and can complete their B.S.N, with two years of 
additional study. 

GOAL 

To provide students with entry level skills and knowledge to function effectively as a 
registered nurse. 




HEALTH INDICATORS - NURSING 




PROGRAM TREND DATA 
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PROGRAM DATA SUMMARY 
PROGRAM: NURSING (ADN) SEMESTER: FALL 1991 



1 . PROGRAM DEMAND 



Total Number of Applicants: 


128 


t'L'H Major Students: 


138 


Total Qualified Applicants: 


52 


FTE Other Students: 


0% 


New Students: 


40 


% Special Population: 


20.0% 


Continuing Students: 


140 






Total Majors: 


180 







Current Work Force: 
Projected Woiic Force: 


7,500 
8,700 


Total Openings/Y ear: 
New Openings/Year: 


1,200 
300 


2. PROGRAM EFFICIENCY 


Classes Taught: 
# of Small Classes: 
Total Semester Hours: 
Total Student Sem. Hours: 


56 

5 

246 
2,072 


Average Class Size: 
% Small Classes: 
% Sem Hr by Lecturer: 
FTE Students/FTE Fac. 


10 
8.9% 
0% 
8.6 


Cost per Student Sem. Hour: 


$231 






3 . PROGRAM OUTCOMES 


Credit Earned Ratio (Major): 90.6 
Credit Earned Ratio (Remedial): NA 
Credit Earned Ratio (Gen. Ed.): 87.5 


Standard = 75.0 
Standaid = 60.0 
Standard = 65.0 




% Retention of New Majors: 


90 


Standard = 50.0 




Degrees Awarded: 
Graduation Rate (Fed): 
Graduation Rate (UH): 
Student Transfer Rate: 


32 

NA 

75% 

14% 






% Grads Employed in Field: 
Licensing Rate: 


98.3% 
85.0% 


Standard = 70.0 




Graduate Satisfaction Rate: 
Leaver Satisfaction Rate: 


4.7 
NA 







PROGRAM ANALYSIS AND RECOMMENDATIONS 

The nursing program has generally performed above the expected levels. There are no 
major problems facing the program at this time. 

An analysis of the student success rate on the licensure examination, which was lower than 
desirable reveals that many of the failures were among students who were completmg the 
program after being admitted as Ucensed practical nurses rather than as fiill two year 

students It is unclear yet whether this is the result of the LPN students bemg out of school 
for some time and therefore not as familiar with the content of the first year courses, ^ 
whether this is a one time problem, or whether there is some other unknown cause. It is 
recommended that an analysis be done of the areas of the t^st on which 
students did poorly and whether that is related to the curriculum sequencmg 
taken by LPN admittees. It is further recommended that this performance 
be monitored in future years and that LPN graduates be encouraged to 
participate in licensure preparation courses. 

Discussion with local hospital personnel indicate that there is beginning to be a slight 
improvement in the severe shortage for registered nurses in the State. It is 
recommended that this trend should be monitored to see if the demand will 
continue to drop in the years to come. 

There is now a significant surplus in the number of qualified students and the number of 
students that can be admitted each term. The qualified students who are turned down tor 
admission after investing one to two years in preparatory course work are likely to be very 
upset It is recommended that a committee be formed to examine the 
admission issues and consider the initiation of a pre-nursing major that 
were provide for guaranteed admission for qualified students. 

The nursing program is applying for initial accreditation from the National League of 
Nursing in 1992. This may result in curricular or other changes bemg recommended m 
response to Uie accreditation report and site visit Final action on tiie accreditatton is 
expected in January 1993. 



